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Abstract 

This paper shows a relationship between two different approximation techniques: the 
Support Vector Machines (SVM), proposed by V. Vapnih (1995), and a sparse ap- 
proximation scheme that resembles the Basis Pursuit De-Noising algorithm (Chen, 
1995; Chen, Donoho and Saunders, 1995). SVM is a technique which can be derived 
from the Structural Push Minimization Principle (Vapnih, 1982) and can be used 
to estimate the parameters of several different approximation schemes, including Ra- 
dial Basis Functions, algebraic/trigonometric polynomials, B-splines, and some forms 
of Multilayer Perceptrons. Basis Pursuit De-Noising is a sparse approximation tech- 
nique, in which a function is reconstructed by using a small number of basis functions 
chosen from a large set (the dictionary). We show that, if the data are noiseless, the 
modified version of Basis Pursuit De-Noising proposed in this paper is equivalent to 
SVM in the following sense: if applied to the same data set the two techniques give 
the same solution, which is obtained by solving the same quadratic programming 
problem. In the appendix we also present a derivation of the SVM technique in the 
frameworh of regularization theory, rather than statistical learning theory, establish- 
ing a connection between SVM, sparse approximation and regularization theory. 
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1 Introduction 

In recent years there has been an increasing interest in approximation techniques that use the 
concept of sparsity to perform some form of model selection. By sparsity we mean, in very 
general terms, a constraint that enforces the number of building blocks of the model to be small. 
Sparse approximation often appears in conjunction with the use of overcomplete or redundant 
representations, in which a signal is approximated as a linear superposition of basis functions 
taken from a large dictionary (Chen, 1995; Chen, Donoho and Saunders, 1995; Olshausen and 
Field, 1996; Daubechies, 1992; Mallat and Zhang, 1993; Coifman and Wickerhauser, 1992). In 
this case sparsity is used as a criterion to choose between different approximating functions 
with the same reconstruction error, favoring the one with the least number of coefficients. The 
concept of sparsity has also been used in linear regression, as an alternative to subset selection, 
in order to produce linear models that use a small number of variables and therefore have greater 
interpretability (Tibshirani, 1994; Breiman, 1993). 

In this paper we discuss the relationship between an approximation technique based on the prin- 
ciple of sparsity and the Support Vector Machines (SVM) technique recently proposed by Vapnik 
(Vapnik, 1995; Vapnik, Golowich and Smola, 1996). SVM is a classification/approximation tech- 
nique derived by V. Vapnik in the framework of Structural Risk Minimization, which aims at 
building "parsimonious" models, in the sense of VC-dimension. Sparse approximation technique 
are also "parsimonious" , in the sense that they try to minimize the number of parameters of the 
model, so it is not surprising that some connections between SVM and sparse approximation 
exist. What is more surprising and less obvious is that SVM and a specific model of sparse 
approximation, which is a modified version of the Basis Pursuit De-Noising algorithm (Chen, 
1995; Chen, Donoho and Saunders, 1995), are actually equivalent, in the case of noiseless data. 
By equivalent we mean the following: if applied to the same data set they give the same solution, 
which is obtained by solving the same quadratic programming problem. While the equivalence 
between sparse approximation and SVM for noiseless data is the main point of the paper, we 
also include a derivation of the SVM which is different from the one given by V. Vapnik, and 
that fits very well in the framework of regularization theory, the same one which is used to derive 
techniques like splines or Radial Basis Functions. 

The plan of the paper is as follows: in section 2 we introduce the technique of SVM in the 
framework of regularization theory (the mathematical details can be found in appendix B). 
Section 3 introduces the notion of sparsity and presents an exact and approximate formulation 
of the problem. In section 4 we present a sparse approximation model, which is similar in spirit 
to the Basis Pursuit De-Noising technique of Chen, Donoho and Saunders (1995), and show 
that, in the case of noiseless data, it is equivalent to SVM. Section 5 concludes the paper and 
contains a series of remarks and observations. Appendix A contains some background material on 
Reproducing Kernel Hilbert Spaces, which are heavily used in this paper. Appendix B contains 
an explicit derivation of the SVM technique in the framework of regularization theory, and 
appendix C addresses the case in which data are noisy. 



2 From Regularization Theory to Support Vector Ma- 
chines 

In this section we briefly sketch the ideas behind the Support Vector Machines (SVM) for regres- 
sion, and refer the reader to (Vapnik, 1995) and (Vapnik, Golowich and Smola, 1996) for a full 
description of the technique. The reader should be warned that the way the theory is presented 
here is slightly different from the way it is derived in Vapnik's work. In this paper we will take a 
viewpoint which is closer to classical regularization theory (Tikhonov and Arsenin, 1977; Moro- 
zov, 1984; Bertero, 1986; Wahba, 1975, 1979, 1990), which might be more familiar to the reader, 
rather than the theory of uniform convergence in probability developed by Vapnik (Vapnik, 1982; 
Vapnik, 1995). A similar approach is described in (Smola and Scholkopf, 1998), although with 
a different formalism. In this section and in the following ones we will need some basic notions 
about Reproducing Kernel Hilbert Spaces (RKHS). For simplicity of exposition we put all the 
technical material about RKHS in appendix (A). Since the RKHS theory is very well developed 
we do not include many important mathematical technicalities (like the convergence of certain 
series, or the issue of semi-RKHS), because the goal here is just to provide the reader with a 
basic understanding of an already existing technique. The rigorous mathematical apparatus that 
we use can be mostly found in chapter 1 of the book of G. Wahba (1990) . 

2.1 Support Vector Machines 

The problem we want to solve is the following: we are given a data set D = {(x 8 -, J/ 8 )}' =1 , obtained 
by sampling, with noise, some unknown function /(x) and we are asked to recover the function 
/, or an approximation of it, from the data D. We assume that the function / underlying the 
data can be represented as: 

oo 

/(x) = £cA(x) + 6 (1) 

71 = 1 

where {^) n (x)}^ 1 is a set of given, linearly independent basis functions, and c n and b are pa- 
rameters to be estimated from the data. Notice that if one of the basis functions <f> n is constant 
then the term b is not necessary. The problem of recovering the coefficients c n and b from the 
data set D is clearly ill-posed, since it has an infinite number of solutions. In order to make 
this problem well-posed we follow the approach of regularization theory (Tikhonov and Arsenin, 
1977; Morozov, 1984; Bertero, 1986; Wahba, 1975, 1990) and impose an additional smoothness 
constraint on the solution of the approximation problem. Therefore we choose as a solution the 
function that solves the following variational problem: 

min#Lf] = C£V( yi - /(x,-)) + U[f\ (2) 

where V(x) is some error cost function that is used to measure the interpolation error (for 
example V(x) = x 2 ) } C is a positive number, $[/] is a smoothness functional and Ti is the set of 
functions over which the smoothness functional $[/] is well defined. The first term is enforcing 
closeness to the data, and the second smoothness, while C controls the tradeoff between these 
two terms. A large class of smoothness functionals, defined over elements of the form (1), can 
be defined as follows: 



oo 2 

71=1 A ™ 

where {A n }^ x is a decreasing, positive sequence. 

That eq. (3) actually defines a smoothness functional can be seen in the following: 

Example: Let us consider a one-dimensional case in which x £ [0,27r], and let us choose 
<t>n(z) = e mx , so that the c n are the Fourier coefficients of the function /. Since the sequence 
{X n }^ D =1 is decreasing, the constraint that $[/] < oo is a constraint on the rate of convergence to 
zero of the Fourier coefficients c n , which is well known to control the differentiability properties 
of /. Functions for which $[/] is small have limited high frequency content, and therefore do 
not oscillate much, so that $[/] is a measure of smoothness. More examples can be found in 
appendix A. 

When the smoothness functional has the form (3) it is easy to prove (appendix B) that, inde- 
pendently on the form of the error function V, the solution of the variational problem (2) has 
always the form: 

i 
/(x) = $>.-#(x,x,-) + & (4) 

8 = 1 

where we have defined the (symmetric) kernel function K as: 

oo 

#( x ; y) = J2 A,i<M x )<My) ( 5 ) 

71 = 1 

The kernel K can be seen as the kernel of a Reproducing Kernel Hilbert Space (RKHS), a concept 
that will be used in section (4). Details about RKHS and examples of kernels can be found in 
appendix A and in (Girosi, 1997). 

If the cost function V is quadratic the unknown coefficients in (4) can be found by solving a 
linear system. When the kernel K is a radially symmetric function eq. (4) describe a Radial 
Basis Functions approximation scheme, which is closely related to smoothing splines, and when 
K is of the form K(x. — y) eq. (4) is a Regularization Network (Girosi, Jones and Poggio, 1995). 
When the cost function V is not quadratic anymore the solution of the variational problem (2) 
has still the form (4) (Smola and Scholkopf, 1998; Girosi, Poggio and Caprile, 1991), but the 
coefficients a 8 - cannot be found anymore by solving a linear system. V. Vapnik (1995) proposed 
to use a particularly interesting form for the function V, which he calls the e-insensitive cost 
function, which we plot in figure (1): 

V(x) = \xL = l° l if M< e (6) 

1 \x\ — e otherwise. 

The e-insensitive cost function is similar to some of the functions used in robust statistics (Huber, 
1981), which are known to provide robustness against outliers. However the function (6) is not 
only a robust cost function, but also assigns zero cost to errors which are smaller then e. In other 
words, according to the cost function \x\ e any function that comes closer than e to the data points 
is a perfect interpolant. In a sense, the parameter e represents, therefore, the resolution at which 
we want to look at the data. When the e-insensitive cost function is used in conjunction with 
the variational approach of (2), one obtains the approximation scheme known as SVM, which 
has the form 



V(x) 



X 



Figure 1: Vapnik's e-insensitive cost function V(x) 



/(x, a, a*) = ^2(a* - a 8 )A(x; x,-) + 6, 



(7) 



i=l 



where a* and a 8 - are some positive coefficients which solve the following Quadratic Programming 
(QP) problem: 



mini?(a*, a) = e^(a* + a,-) - ^ 2/*(«* - «0 + 77 Z ( a * ~ a «')( a j ~ «j)^( x ^ x j)> ( 8 ) 



subject to the constraints 



< a*, a < C 

EL 



[OL; 



OLi 



(9) 



OLiOL- 







Vz = 1, 



/ 



Notice that the parameter b does not appear in the QP problem, and we show in appendix (B) 
that it is determined from the knowledge of a and a*. It is important to notice that it is possible 
to prove that the last of the constraints above (a 8 a* = 0) is automatically satisfied by the solution 
and it could be dropped from the formulation. We include this constraint just because it will be 
useful in section 4. 

Due to the nature of this quadratic programming problem, only a number of coefficients a* — eti 
will be different from zero, and the input data points x 8 - associated to them are called support 
vectors. The number of support vectors depends on both C and e. The parameter C weighs the 
data term in functional (2) with respect to the smoothness term, and in regularization theory is 
known to be related to the amount of the noise in the data. If there is no noise in the data the 
optimal value for C is infinity, which forces the data term to be zero. In this case SVM will find, 
among all the functions which have interpolation errors smaller than e, the one that minimizes 
the smoothness functional $[/]. The parameters C and e are two free parameters of the theory, 
and their choice is left to the user, as well as the choice of the kernel K, which determines the 
smoothness properties of the solution and should reflect prior knowledge on the data. For certain 
choices of K some well known approximation schemes are recovered, as shown in table (I). We 
refer the reader to the book of Vapnik (1995) for more details about SVM, and for the original 
derivation of the technique. 



Kernel Function 


Approximation Scheme 


K(x;y) = exp(- x - y 2 ) 


Gaussian RBF 


tf(x;y) = (l + x-y) d 


Polynomial of degree d 


A"(x; y) = tanh(x • y — 0) 


(only for some values of 6) 
Multi Layer Perceptron 


K(x;y) = B 2n (x-y) 


B-splines 


J\{x,y — . ( ) 

sm ^ — j-^- 


Trigonometric polynomial of degree d 



Table 1: Some possible kernel functions and the type of decision surface they define. The last 
two kernels are one-dimensional: multidimensional kernels can be built by tensor products of 
one-dimensional ones. The functions B n are piecewise polynomials of degree n, whose exact 
definition can be found in (Schumaker, 1981) 



3 Sparse Approximation 

In recent years there has been a growing interest in approximating functions using linear su- 
perpositions of basis functions selected from a large, redundant set of basis functions, called 
dictionary. It is not the purpose of this paper to discuss the motivations that lead to this ap- 
proach, and refer the reader to (Chen, 1995; Chen, Donoho and Saunders, 1995; Olshausen and 
Field, 1996; Harpur and Prager, 1996; Daubechies, 1992; Mallat and Zhang, 1993; Coifman and 
Wickerhauser, 1992) for further details. A common aspects of these technique is that one seeks 
an approximating function of the form: 

n 

/( x ; a ) = ^<w(x) (io) 

8 = 1 

where ip = {( J c 8 (x)}™_ 1 is a fixed set of basis functions that we will call dictionary. If n is very 
large (possibly infinite) and ip is not an orthonormal basis (for example it could be a frame, or 
just a redundant, finite set of basis functions) it is possible that many different sets of coefficients 
will achieve the same error on a given data set. A sparse approximation scheme looks, among all 
the approximating functions that achieve the same error, for the one with the smallest number 
of non-zero coefficients. The sparsity of an approximation scheme can also be invoked whenever 
the number of basis functions initially available is considered, for whatever reasons, too large 
(this situation arises often in Radial Basis Functions applied to a very large data set). 
More formally we say that an approximating function of the form (10) is sparse if the coefficients 
have been chosen so that they minimize the following cost function: 

n n 

E[*,t] = H/(x)-E^^(x)||i 2 + A(^6r (11) 

8=1 8=1 

where {^8'}™ =1 is a set of binary variables, with values in {0, 1} ,|| • \\l 2 is the usual L 2 norm, and 
p is a positive number that we set to one unless otherwise stated. It is clear that, since the L 
norm of a vector counts the number of elements of that vector which are different from zero, the 
cost function above can be replaced by the cost function: 

n 

£[a] = ||/(x)-£w(*)HL + *NL (12) 

8 = 1 

The problem of minimizing such a cost function, however, is extremely difficult because it in- 
volves a combinatorial aspect, and it will be impossible to solve in practical cases. In order to 
circumvent this problem, approximated versions of the cost function above have been proposed. 
For example, in (Chen, 1995; Chen, Donoho and Saunders, 1995) the authors use the L\ norm 
as an approximation of the L 0} obtaining an approximation scheme that they call Basis Pursuit 
De-Noising. In related work, Olshausen and Field (1996) enforce sparsity by considering the 
following cost function: 

n n 

£[a] = ||/(x) - £ WMHL + e£SK-) (13) 

8=1 j=l 

where the function S was chosen in such a way to approximately penalize the number of non-zero 
coefficients. Examples of some the choices considered by Olshausen and Field (1996) are reported 
in table (2). 



S(x) 


\x\ 


— exp(— x 2 ) 


log(l + x 2 ) 



Table 2: Some choices for the penalty function S in eq. (13) considered by Olshausen and Field 
(1996). 

In the case in which S(x) = \x\, that is the Basis Pursuit De- Noising case, it is simple to see 
how the cost function (13) is an approximated version of the one in (11). In order to see this, 
let us allow the variables £,- to assume values in { — 1,0,1}, so that the cost function (11) can be 
rewritten as 



i?[a,^ = ||/(x)-^6«^(x)||| 2 + A^|6 



(14) 



8 = 1 



8 = 1 



If we now let the variables £,- assume values over the all real line, and assuming that the coefficients 
a{ are bounded, it is clear that the coefficients a 8 - are redundant, and can be dropped from the 
cost function. Renaming the variables £,- as a,-, we then have that the approximated cost function 



is 



£[a] = ||/(x)-;>>^(x)||i 2 +A||a|| Ll 



(15) 



8=1 



which is the one proposed in the Basis Pursuit De-Noising method of Chen, Donoho and Saunders 
(1995). 

4 An Equivalence Between Support Vector Machines 
and Sparse Coding 

The approximation scheme proposed by Chen, Donoho and Saunders, (1995) has the form de- 
scribed by eq. (10), where the coefficients are found by minimizing the cost function (15). We 
now make the following choice for the basis functions ipf. 



<fi(x) = A"(x;x 8 ) Vz = 1, 



/ 



where A (x; y) is the reproducing kernel of a Reproducing Kernel Hilbert Space (RKHS) Ti (see 
appendix A) and {(x 8 , j/ 8 )}' =1 is a data set which has been obtained by sampling, in absence 
of noise, the target function /. We make the explicit assumption that the target function / 
belongs to the RKHS Ti. The reader unfamiliar with RKHS can think of Ti as a space of smooth 
functions, for example functions which are square integrable and whose derivatives up to a certain 
order are also square integrable. The norm ||/||^ in this Hilbert space can be thought as a linear 
combination of the L 2 norm of the function and the L 2 norm of its derivatives (the specific degree 
of smoothness and the linear combination depends on the specific kernel K). It follows from eq. 
(10) that our approximating function is: 



r(x) = /(x;a) = $>tf(x;x,-) (16) 

8 = 1 

This model is similar to the one of SVM (eq. 7), except for the constant 6, and if A^(x;y) = 
G(||x — y||), where G is a positive definite function, it corresponds to a classical Radial Basis 
Functions approximation scheme (Micchelli, 1986; Moody and Darken, 1989; Powell, 1992). 
While Chen et al., in their Basis Pursuit De-Noising method, measure the reconstruction error 
with an L 2 criterion, we measure it by the true distance, in the Ti norm, between the target 
function / and the approximating function /*. This measure of distance, which is common in 
approximation theory, is better motivated than the L 2 norm because it not only enforces closeness 
between the target and the model, but also between their derivatives, since || • ||ft is a measure 
of smoothness. We therefore look for the set of coefficients a that minimize the following cost 
function: 

£[a] = ^ll/(x) " J>tf (x;x,-)|& + e|Mk (17) 

8 = 1 

where || • ||ft is the standard norm in Ti. We consider this to be a modified version of the Basis 
Pursuit De-Noising technique of Chen (1995) and Chen, Donoho and Saunders (1995). 
Notice that it looks from eq. (17) that the cost function E cannot be computed because it 
requires the knowledge of / (in the hrst term). This would be true if we had || • \\l 2 instead of 
|| • ||ft in eq. (17), and it would force us to consider the approximation: 

ii/(x)-r(x)iii 2 ^yE(j/ 8 -r(x 8 )) 2 (is) 

However, because we used the norm || • ||ft, we will see in the following that (surprisingly) no 
approximation is required, and the expression (17) can be computed exactly, up to a constant 
(which is obviously irrelevant for the minimization process). 

For simplicity we assume that the target function / has zero mean in 7i, which means that its 
projection on the constant function <?(x) = 1 is zero: 

</,l>H=0 

Notice that we are not assuming that the function <?(x) = 1 belongs to 7i, but simply that the 
functions that we consider, including the reproducing kernel K , have a finite projection on it. In 
particular we normalize K in such a way that < l,A^(x;y) >n= 1- We impose one additional 
constraints on this problem: 

• We want to guarantee that the approximating function /* has also zero mean in Ti: 

<r,i>ft=o (19) 

Substituting eq. (16) in eq. (19), and using the fact that K has mean equal to 1, we see that 
this constraint implies that: 

i 
5>.- = ° • (20) 

8 = 1 



We can now expand the cost function E of equation (17) as 



E i a ] = gll/llw ~ Z) a « < /( x ), A"(x;x 8 ) >h +2 Z a i a i < A"(x;x 8 ), A^xjXj) > H +e^ \a % \ 

8 = 1 ? J = 1 ? = 1 

Using the reproducing property of the kernel K we have: 



< /(x), #(x;x,-) >h= /(x,-) = j/ 8 

< A"(x;x 8 ),A"(x;Xj) > H = A"(x 8 ;x^ 



(21) 
(22) 



Notice that in eq. (21) we explicitly used the assumption that the data are noiseless, so that we 
know the value j/ 8 - of the target function / at the data points x 8 -. We can now rewrite the cost 
functions as: 



1 



1 






(23) 



We now notice that the L\ norm of a (the term with the absolute value in the previous equation), 
can be rewritten more easily by decomposing the vector a in its "positive" and "negative" parts 
as follows: 

a = a + — a~ a + , a~ > 0, afa~ = Vz = 1, . . . , /. 



Using this decomposition we have 



k = J2( a i + a i 



(24) 



8 = 1 



Disregarding the constant term in ||/||^ and taking in account the constraint (20), we conclude 
that the minimization problem we are trying to solve is equivalent to the following quadratic 
programming (QP) minimization problem: 



Problem 4.1 Solve: 



mm 

a ,a~ 



Z ,a »'' ~~ a 



1 



I 



I 



8 = 1 



+ " Vv + 2 J2 ( a t - a 8 )( a t - a j ) K (^ Xj) + e J2( a i + a 8 



M=l 



8=1 



subject to the constraints: 



(25) 



8l . 3. 


>o 






E(=iK + - 


-a") =0 






«8 + «8 


= Vz = 


1,- 


.,/ 



(26) 



If we now rename the coefficients as follows: 



«8 



«8 



we notice that the QP problem defined by equations (25) and (26) is the same QP problem that 
we need to solve for training a SVM with kernel K (see eq. 8 and 9) in the case in which the 
data are noiseless. In fact, as we argued in section 2.1, the parameter C of a SVM should be set 
to infinity when the data are noiseless. Since the QP problem above is the same QP problem of 
SVM, we can use the fact that the constraint a 8 a* = is automatically satisfied by the SVM 
solution (see appendix B) to infer that the constraint afa~ = is also automatically satisfied in 
the problem above, so that it does not have to be included in the QP problem. Notice also that 
the constant term b which appears in (7) does not appear in our solution. We argue in appendix 
B that for most commonly used kernels K this term is not needed, because it is already implicitly 
included in the model. We can now make the following: 

Statement 4.1 When the data are noiseless, the modified version of Basis Pursuit De-Noising 
of eq. (17), with the additional constraint (19), gives the same solution of SVM, and the solution 
is obtained by solving the same QP problem of SVM. 

As expected, the solution of the Basis Pursuit De-Noising is such that only a subset of the data 
points in eq. (16) has non-zero coefficients, the so-called support vectors. The number of support 
vectors, that is the degree of sparsity, is controlled by the parameter e, which is the only free 
parameter of this theory. 

5 Conclusions and remarks 

In this paper we showed that, in the case of noiseless data, SVM can be derived without using any 
result from VC theory, but simply enforcing a sparsity constraint in an approximation scheme of 
the form 

i 
/( x ; a ) = J2 av-K^x; x,-) 

8 = 1 

together with the constraint that, assuming that the target function has zero mean, the approxi- 
mating function should also have zero mean. This makes a connection between a technique such 
as SVM, which is derived in the framework of Structural Risk Minimization, and Basis Pursuit 
De-Noising, a technique which has been proposed starting from the principle of sparsity. Some 
observations are in order: 

• This result shows that SVM provide an interesting solution to an old-standing problem: the 
choice of the centers for Radial Basis Functions. If the number of data points is very large 
we do not want to place one basis function at every data point, but rather at a (small) 
number of other locations, called "centers". The choice of the centers is often done by 
randomly choosing a subset of the data points. SVM provides a subset of the data points 
(the support vectors) which is "optimal" in the sense of the trade-off between interpolation 
error and number of basis functions (measured in the L\ norm). SVM can be therefore seen 
as a "sparse" Radial Basis Functions in the case in which the kernel is radially symmetric. 

• One can regard this result as an additional motivation to consider sparsity as an "interest- 
ing" constraint. In fact, we have shown here that, under certain conditions, sparsity leads 
to SVM, which is related to the Structural Risk Minimization principle, and is extremely 
well motivated in the theory of uniform convergence in probability. 

10 



• The result holds because in both this and Vapnik's formulation the cost function contains 
both an "_L 2 -type" and an "i^-type" norm. However, the Support Vector method has an 
"-Li-type" norm in the error term, and an L 2 norm in the "regularization" term, while the 
cost function (17) we consider has an "_L 2 -type" norm in the error term and an L\ norm in 
the "regularization" term. 

• This results holds due to the existence of the reproducing property of the RKHS. If the 
norm || • ||^ were replaced by the standard L 2 norm or any other Sobolev norm the cost 
function would contain the scalar product in L 2 between the unknown function / and the 
kernel A^(x;x 8 ), and the cost function could not be computed. If we replace the RKHS 
norm with the training error on a data set {(x 8 -, j/i)}' =1 (as in Basis Pursuit De-Noising) the 
cost function could be computed, but it would lead to a different QP problem. Notice that 
the cost function contains the actual distance between the approximating and the unknown 
function, which is exactly the quantity that we want to minimize. 

• As a side effect, this paper provides a derivation of the SVM algorithm in the framework 
of regularization theory (see appendix B). The advantage of this formulation is that it is 
particularly simple to state, and it is easily related to other well known techniques, such 
as smoothing splines and Radial Basis Functions. The disadvantage is that it hides the 
connection between SVM and the theory of VC bounds, and does not make clear what 
induction principle is being used. When the output of the target function is restricted to 
be 1 or -1, that is we consider a classification problem, Vapnik shows that SVM minimize 
an upper bound on the generalization error, rather than minimizing the training error 
within a fixed architecture. Although this is rigorously proved only in the classification 
case, this is a very important property, that makes SVM extremely well founded from the 
mathematical point of view. This motivation, however, is missing when the regularization 
theory approach is used to derive SVM. 

• The equivalence between SVM and sparsity has only been shown in the case of noiseless 
data. In order to maintain the equivalence in the case of noisy data, one should prove that 
the presence of noise in the problem (17) leads to the additional constraint a* } a < C as 
in SVM, where C is some parameter inversely related to the amount of noise. In appendix 
C we sketch a tentative solution to this problem. This solution, however, is not very 
satisfactory because is purely formal, and it does not explain what assumptions are made 
on the noise in order to maintain the equivalence. 

Acknowledgments I would like to thank T. Poggio and A. Verri for their useful comments and B. 
Olshausen for the long discussions on sparse approximation. 

A Reproducing Kernel Hilbert Spaces 

In this paper, a Reproducing Kernel Hilbert Space (RKHS) (Aronszajn, 1950) is defined a Hilbert 
space of functions defined over some domain C R d with the property that, for each x £ 0, the 
evaluation functionals JF X defined as 

JxLf] = /(x) V/ £ H 
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are linear, bounded functionals. It can be proved that to each RKHS Ti it corresponds a positive 
definite function A^(x,y), which is called the reproducing kernel of Ti. The kernel of Ti has the 
following reproducing property: 

/(x) =< /(y), A(y; x) > n VftH (27) 

where < •,• >^ denotes the scalar product in Ti. The function K acts in a similar way to 
the delta function in L 2} although L 2 is not a RKHS (its elements are not necessarily defined 
pointwise). Here we sketch a way to construct a RKHS, which is relevant to our paper. The 
mathematical details (such the convergence or not of certain series) can be found in the theory of 
integral equations (Hochstadt, 1973; Cochran, 1972; Courant and Hilbert, 1962), which is very 
well established, so we do not discuss them here. In the following we assume that = [0, l] d for 
simplicity. The main ideas will carry over to the case fl = R d , although with some modifications, 
as we will see in section (A. 2). 

Let us assume that we find a sequence of positive numbers X n and linearly independent functions 
</>„(x) such that they define a function A^(x;y) in the following way 1 : 



oo 



A(x;y) = ^A^4x)^(y) (28) 

71 = 1 

where the series is well defined (for example it converges uniformly). A simple calculation shows 
that the function K defined in eq. (28) is positive semi-definite. Let us now take as Hilbert 
space the set of functions of the form 

oo 

/(x) = £^(x) (29) 



71 = 1 



in which the scalar product is defined as: 



oo 



< X! C ™ < M X )>X! ^ < M X ) >n= Yl ~T JL ( 30 ) 

71 = 1 71 = 1 71=1 n 

Assuming that all the evaluation functionals are bounded, it is now easy to check that such an 
Hilbert space is a RKHS with reproducing kernel given by A (x; y). In fact we have 

< /(x), A(x; y) > n = £ 2^M_ = £ CnMy) = /(y) . 

71 = 1 n 71=1 

We conclude that it is possible to construct a RKHS whenever a function K of the form (28) is 
available. The norm in this RKHS has the form: 

°o 2 

ll/llw = £f (31) 

71=1 An 

It is well known that expressions of the form (28) actually abound. In fact, it follows from 
Mercer's theorem (Hochstadt, 1972) then any function A^(x;y) which is the kernel of a positive 
operator 2 in L 2 (fl) has an expansion of the form (28), in which the cf>i and the A 8 - are respectively, 



^^When working with complex functions <^„(x) this formula should be replaced with A'(x;y) = 

2 We remind the reader that positive operators in L'j are self-adjoint operators such that < Kf, f > > for 
all/GL 2 . 
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the orthogonal eigenfunctions and the positive eigenvalues of the operator corresponding to K. In 
(Stewart, 1976) it is reported that the positivity of the operator associated to K is equivalent to 
the statement that the kernel K is positive definite, that is the matrix Kij = A^(x 8 -; Xj) is positive 
definite for all choices of distinct points x 8 -. Notice that a kernel K could have an expansion of 
the form (28) in which the <f> n are not necessarily its eigenfunctions. 

The case in which = R d is similar, with the difference that the eigenvalues may assume any 
positive value, so that there will be a non-countable set of orthogonal eigenfunctions. In the 
following section we provide a number of examples of these different situations, that also show 
why the norm |/||^ can be seen as a smoothness functional. 

A.l Examples: RKHS over [0, 2tt] 

Here we present a simple way to construct meaningful RKHS of functions of one variable over 

[0,27r]. In the following all the normalization factors will be set to 1 for simplicity. 

Let us consider any function K(x) which is continuous, symmetric, periodic, and whose Fourier 

coefficients X n are positive. Such a function can be expanded in a uniformly convergent Fourier 

series: 

oo 

K(x) = ^A n cos(nx) . (32) 

n=0 

An example of such a function is 

1 1-h 2 



K(x) = l + J2h 



cos (nx 



n=1 2Trl-2hcos(x) + h 2 

where h £ (0, 1). 

It is easy to check that, if (32) holds, then we have: 

oo oo 

K(x — y) = 1 + 22 ^n sm(nx) sm(ny) + Jj \ n cos(nx) cos(ny) (33) 

n=l n=l 

which is of the form (28) in which the set of orthogonal functions <f> n has the form: 



{<f>i(x)}°^ = (1, sin(x), cos(x), sin(2x), cos(2x), . . . , sin(nx), cos(nx] 



i • • • > 



Therefore, given any function K which is continuous, periodic and symmetric we can then define 
a RKHS Ti over [0, 2tt] by defining a scalar product of the form: 



, r . _ \~^ Jn9n ' Jn9 

<f,9 >H= 2^- 



n=0 ^ n 

where we use the following symbols for the Fourier coefficients of a function /: 

fn =< f, cos(nx) > , f s n =< f, sm(nx) > 

The functions in Ti are therefore functions in iv2([0,27r]) whose Fourier coefficients satisfy the 
following constraint: 

2 \~^ \Jn) T \Jn) - , ^^ (qa\ 

h = 2s 1 < +°° i 34 ) 
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Since the sequence X n is decreasing, the constraint that the norm (34) has to be finite can be seen 
as a constraint on the rate of decrease to zero of the Fourier coefficients of the function /, which 
is known to be related to the smoothness properties of /. Therefore, choosing different kernels 
K is equivalent to choose RKHS of functions with different smoothness properties, and the norm 
(34) can be used as the smoothness functional $[/] in the regularization approach sketched in 
section 2. The relationship between the kernel K and the smoothness properties of the functions 
in the corresponding RKHS will become more clear in the next section, where we discuss the 
extension of this approach to the infinite domain = R d . 

A. 2 Examples: RKHS over R d 

When the domain over which we wish to define a RKHS becomes the whole space R d most of 
the results of the previous section still apply, with the difference that the spectrum of K becomes 
(usually) the whole positive axis, and it is not countable anymore. 

For translation invariant kernels, that is positive definite functions of the form K(x. — y), the 
following decomposition holds: 

K(x - y) = / dsK(s)e iB - x e- iB - y (35) 

JR d 

Equation (35)is the analog of (28) over an infinite domain, and one can go from the case of 
bounded to the case of = R by the following substitutions: 

n =>- s 

\ n =>■ K(s) 

<f> n (x.) => e 8S ' x 

E^=i => JRd ds 

We conclude then that any positive definite function of the form K(x. — y) defines a RKHS over 
R d by defining a scalar product of the form 



J(s)ff*(s 

k(s 

The reproducing property of K is easily verified: 



<f , g>H ^i ds f^m ( 36) 

J A(s) 



</(,U(»-y)> = / ^"y- =/(y) 

J K (s) 

and the RKHS becomes simply the subspace of Z 2 (i? d ) of the functions such that 



n = / ds -J77— < +°° 37 

J K (s) 

Functionals of the form (37) are known to be smoothness functionals. In fact, the rate of decrease 
to zero of the Fourier transform of the kernel will control the smoothness property of the function 
in the RKHS. Consider for example, in one dimension, the kernel K(x) = e~' x ', whose Fourier 
Transform is K(s) = (I + s 2 ) -1 . The RKHS associated to this kernel contain functions such 

2 n = J ds ^ = \\f\\l + \\f\\l < - 
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This is the well known Sobolev space H^ 1 , where we denote by W™ the set of functions whose 
derivatives up to order m are in L 2 (Yosida, 1974). Notice that the norm induced by the scalar 
product (36) is the smoothness functional considered by Girosi, Jones and Poggio (1995) in their 
approach to regularization theory for function approximation. This is not surprising, since RKHS 
have been known to play a central role in spline theory (Wahba, 1990). Notice also that in spline 
theory one actually deals with semi-RKHS, in which the norm || • ||^ has been substituted with a 
semi-norm. Semi-RKHS share most of the properties of RKHS, but their theory becomes a little 
more complicated because of the null space of the semi-norm, which has to be taken in account. 
Details about semi-RKHS can be found in (Wahba, 1990). 

A. 3 Finite Dimensional RKHS 

When the set of basis functions cf>i has finite cardinality iV, the construction of a RKHS sketched 
in the previous section (eq. 1 and 30) is always well defined, as long as the basis functions are 
linearly independent. Notice that the functions cf>i do not have to be orthogonal, and that they 
will not be the eigenfunctions of K. It is interesting to notice that in this case we can define a 
different set of basis functions </>;, which we call the dual basis functions, with some interesting 
properties. The dual basis functions are defined as: 



N 

/l/ t 



<Mx) = £ M^Vi(x) (38) 

where M _1 is the inverse of the matrix M: 

Mij=<<j>i,<j>j> (39) 

and the scalar product is taken in L 2 . It is easy to verify that, for any function of the space, the 
following identity holds: 

N N 

/(x) = £</,&> &(x) = £</,&> <Mx) (40) 

8 = 1 8 = 1 

where the second part of the identity comes from the fact that the dual basis of the dual basis is 
the original basis. From here we conclude that, for any choices of positive A,-, the set of functions 
spanned by the functions cf>i form a RKHS whose norm is 

JV r 7 2 

11/11^ E < i > (41) 

8 = 1 ' 

Notice that while the elements of the (dual) basis are not orthogonal to each other, orthogonality 
relationships hold between elements of the basis and the elements of the dual basis: 

<&,<p J >=S lJ (42) 

As a consequence, it is also possible to show that, defining the dual kernel as: 

N 

A>;y) = ]TA 8 <Mx)<My) (43) 

8 = 1 

the following relationships hold: 
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dy K(-x;y)<f>i(y) = A,-^ t -(x) 
dy K(x;y)<f>i(y) = A,-</>,-(x) 

A. 3.1 A RKHS of polynomials 

In this section we present a particular RKHS, which allows us to partially answer to an old stand- 
ing question: is it possible to derive, in the framework of regularization theory, an approximation 
scheme of the form: 

n 

/(x) = $><t(x-w,- + 0) (44) 

8 = 1 

where a is some continuous, one dimensional function? Connections between approximation 
schemes of the form (44) and regularization theory have been presented before (Girosi, Jones, 
and Poggio, 1995), but always involving approximation or extension of the original regularization 
theory framework. A positive answer to the previous question is given by noticing that it is 
possible to define a RKHS whose kernel is 

A(x;y) = (l + x-y) d 
where d is any integer. In fact, the kernel K above has an expansion of the form: 

N 

A(x; y) = J2 KM*)My) (45) 

71 = 1 

where the <f> n are the monomials of degree up to d, which constitutes a basis in the set of 
polynomials of degree d, and the A 8 - are some positive numbers. The kernel above therefore can 
be used to give the structure of RKHS to the set of polynomials of degree d (in arbitrary number 
of variables), and the norm defined by eq. (41) can be used a smoothness functional in the 
regularization theory approach (see appendix B) to derive an approximating scheme of the form: 

/(x) = ^c 8 (l + x-x 8 ) d 

8 = 1 

which is a special case of eq. (44). The fact that the kernel K(x.;y) = (1 + x • y) d has an 
expansion of the form (45) had been reported by Vapnik (1995). Here we give some examples, 
from which the reader can infer the general case. 
Let us start from the one-dimensional case, where one chooses: 

X n = I J (f) n (x) = x n n = 0, . . . , d . 
It is now easy to see that 

k(x; y ) = J2 KM*)My) = E { t) ( x y) n = ( l + x v) d 

71 = 71 = 



n 



A similar result, although with a more complex structure of the coefficients A n , is true in the 
multivariate case. For example in two variables we can define: 
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{<^-(x)}f =1 = (l,x 1 ,x 2 ,x 1 x 2 ,xl,xl) 
In this case it is easy to verify that: 



tf(x; y) = J2 KM*)My) = (1 + x • y) 2 (46) 



71 = 1 



The same kernel can be obtained in 3 variables by choosing: 

lrn x )li=i = (1? x ~v-> x< 2-> x 3-, x n x 2-> x ?>i x \ x ii X\X^, x 2 x^) 

{Xi}}^ = (1, y/2, V2 , V2 , 1, 1, 1, V2 , V2 , v^ ) 
In 2 variables, we can also make the following choice: 

\ Yi v X //?'=l — \ 1- 1 X\^ X2i X yi X 2i X\X2i XyX2i X\X 2i X-^ i % 2 ) 

{A,-}- = i = (1, V3 , V3 , V3 , V3 , V6, V3 , V3 , 1 , 1) 
and it is easy to see that: 

9 

#(x; y) = J2 KMx)My) = (1 + x • y) 3 

71 = 1 

While it is difficult to write a closed formula for the coefficients A 8 - in the general case, it is 
obvious that such coefficients can always be found. This, therefore, leaves unclear what form of 
smoothness is imposed by using the norm in this particular RKHS, and shows that this example 
is quite an academic one. A much more interesting case is the one in which the function a in 
eq. (44) is a sigmoid or some other activation function. If it were possible to find an expansion 
of the form: 

oo 

cr(x • y + 9) = Y, X n (f) n (x)(f) n (y) 

71 = 1 

for some fixed value of 6, this would mean that a scheme of the form 

71 

/(x) = J^c^i*-^ + 0) 

8 = 1 

could be derived in a regularization theory framework. Vapnik (1995) reports that if a(x) = 
tanh(x) the corresponding kernel is positive definite for some values of 6, but the observation is 
of experimental nature, so that we do not know what the A 8 - or the <f> n are, and therefore we do 
not what kind of functions the corresponding RKHS contains. 
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B Derivation of the SVM Algorithm 
B.l Generalities on Regularization Theory 

Let us look more closely at the solution of the variational problem (2): 

min#Lf] = C£V(yi - /(x,-)) + U[/] 

fen 8=1 2 

We assume that Ti is a RKHS with kernel K and that the smoothness functional $[/] is: 

*[/] = ll/llw 
This is equivalent to assume that the functions in 7i have a unique expansion of the form: 

CO 

/( x ) = J2 c «<M x ) 

7i=l 

and that their norm is: 

oo 2 
2 *~^ c ^ 



H — Z^ 



71=1 ^ n 



In this derivation we do not have the coefficient b which appears in (1), since we argued before 
that if one of the cf>i is constant, which is usually the case, this term is not necessary. 
We can think of the functional H[f] as a function of the coefficients c n . In order to minimize 
H[f] we take its derivative with respect to c n and set it equal to zero, obtaining the following: 



' c 



\r 



~ C £ V'(y t - /(xO)^„(xO + ^ = (47) 

8 = 1 

Let us now define the following set of unknowns: 



a % = CV'(y % - /(x,-)) 
Using eq. (47) we can express the coefficients c n as a function of the a,-: 



C r , 



K^2 ai<f> n (xi) 



8 = 1 



The solution of the variational problem has therefore the form: 

oo oo / / 

/(x) = Y^ c 7i^7i(x) = Yl Yl aiK(f>n(*i)(f>n(x) = Y a *^( x ; X ( 48 ) 

71 = 1 71 = 1 8 = 1 8 = 1 

where we have used the expansion (28). This shows that, independently of the form of V, the 
solution of the regularization functional (2) is always a linear superposition of kernel functions, 
one for each data point. The cost function V affects the computation of the coefficients a,-. In 
fact, plugging eq. (48) back in the definition of the a 8 - we obtain the following set of equations 
for the coefficients a,-: 
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a t = CV {yi-^2 Kt 3 a 3 ' « = 1, • • • , ^ 

where we have defined Kij = A^(Xj-;xj). In the case in which V(x) = x 2 we obtain the standard 
regularization theory solution (see Girosi, Jones and Poggio, 1995 for an alternative derivation): 

(K + 7/) a = y 
where we have defined 7 = ^. 

B.2 The SVM algorithm in the Regularization Theory Framework 

Following Vapnik (1995) we now consider the case of the e-insensitive cost function V(x) = \x\ e . 
In this case the approach sketched above is problematic because V is not differentiable at x = e 
(although it still makes sense everywhere else). In order to make our notation consistent with 
Vapnik's one, we have to modify slightly the model proposed in the previous section. Vapnik 
explicitly takes into account an offset in the model, so that equation (1) is replaced by 



/(x) = £c^(x) + & (49) 

71 = 1 

The smoothness functional remains unchanged (so that the smoothness does not depend on b): 

00 



00 2 



71 = 1 ^ n 

Also, we scale the functional in (2) of a factor ^ = C, obtaining the following variational 
problem: 

mmH[f\ = C'£\y i -f(x i )\ c + U[f\ 

Since it is difficult to deal with the function V(x) = \x\ e , the problem above is replaced by the 
following equivalent 3 problem, in which an additional set of variables is introduced: 

,nin H[f] = Cj2&+ CD + l<S>lf] (50) 



subject to 

/(x,-)-J/v < e + & i = l,..., I 

J/,--/(x,-) < e + g i = l,..., I 

6- > i = i,. ..,1 {0l) 

Ci > z = l,. ..,1 

The equivalence of the variational problem is established just noticing that in the problem above 
a (linear) penalty is paid only when the absolute value of the interpolation error exceeds e, which 
correspond to Vapnik's e-insensitive cost function. Notice that when of the two top constraints is 



3 By equivalent we mean that the function that minimizes the two functional is the same 
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satisfied with some non-zero £,- (or £*), the other is automatically satished with a zero value for 
C* ( or £*')• ^ n or( ier to solve the constrained minimization problem above we use the technique 
of Lagrange multipliers. The Lagrangian corresponding to the problem above is: 

£(/,£,£*; «, a*, r,r*) = C£(& + £) + £*[/] + E a ?(w " /(*) " e " £) + 

8=1 8=1 

+ £ a,-(/(x,-) -yi-e- &) - £(r,-& + rU!) (52) 

8=1 8=1 

where a,a*,r,r* are positive Lagrange multipliers. The solution of the constrained variational 
problem above is now obtained by minimizing the Lagrangian (52) with respect to / (that is 
with respect to the c n and to &), £ and ^* and maximizing (in the positive quadrant) with respect 
to a } a* } r, r*. Since the minimization step is now unconstrained, we set to zero the derivatives 
with respect to c n , &, £ and £*, obtaining: 



dc n 



K^2(a* - ai)<f> n (xi) 



8=1 



<9£ „ 



8 = 1 



dC 



=> r n = C - a n 
=>- r* = C — a* 



Substituting the expression for the coefficients c n in the model (49) we then conclude that the 
solution of the problem (50) is a function of the form 

/(x) = $>,--a.-)tf(x;x,-) + & (53) 

8 = 1 

Substituting eq. (53) in the Lagrangian, we obtain an expression that should now be maximized 
(in the positive quadrant) with respect to a } a* , r, r*, with the additional constraints listed above. 
Noticing that the relationship between r n (r*) and ct n (a*) implies that a < C and a* < C, 
and minimizing —C rather than maximizing C, we now obtain the following QP problem: 



Problem B.l 



1 ' 



min C(a, a*) = e ^(a* + a,-) - J2 Vi( a i ~ a i) + 9 J2 ( a * ~ a 0( a j ~ a j) K (^ x 



a,a 



8=1 8=1 ? j=i 



v j;5 



subject to the constraints 



< a*, a < C 

1 
J2(a* ~ «8') = 



8 = 1 
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This is the QP problem that has to be solved in order to compute the SVM solution. It is useful 
to write and discuss the Kuhn- Tucker conditions: 

a,-(/(x,-) - y % - e - &) = i = 1, . . . , / 

a * i (y i -f(x i )-e-&) = i = l,...,l 

(C-a,-)6- = i = l,...,l 

The input data points x 8 - for which a 8 - or a* are different from zero are called support vectors. 
Few observations are in order: 

• The Lagrange multipliers a 8 - and a* cannot be simultaneously different from zero, so that 
the constraint a 8 a* = holds. 

• The support vectors are those data points x 8 - at which the interpolation error is either 
greater or equal to e. Points at which the interpolation error is smaller than e are never 
support vectors, and do not enter in the determination of the solution. Once they have 
been found, they could be removed from the data set, and if the SVM were run again on 
the new data set the same solution would be found. 

• Any of the support vectors for which < a.{ < C (and therefore £,- = 0) can be used to 
compute the parameter b. In fact, in this case it follows from the Kuhn- Tucker conditions 
that: 

i 
/(xi) = Y, «j A(x 8 ; x j) + b = yi + e 

(a similar argument holds for the a*). 

• If e = then all the points become support vectors; 

• Because of the constraint a 8 a* = 0, defining 

a = a* — ol 

and using eq. (24) the QP problem B.I can be written as follows: 
Problem B.2 

subject to the constraints 

-C <a,<C 
a- 1 = 

Important note: Notice that if one the basis functions cf>i is constant, then the parameter b in 
(49) could be omitted. The RKHS described in appendix A all have this property. 
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min£'*[a] = eUaH^ — a • y -\ — a • A"a 



C Noisy case: an equivalence? 

It is natural to ask whether the result of this paper can be extended to the case of noisy data. I 
will sketch here an argument to show that there is still a relationship between SVM and sparse 
approximation, when data are noisy, although the relationship is much less clear. In the presence 
of additive noise we have 

/(x,-) = yi + Si , 

where j/ 8 - are the measured value of /, and Si are random variables with unknown probability 
distribution. Substituting j/ 8 - with j/ 8 - + Si in eq. (23), disregarding the constant term in ||/||^, 
and defining 

ill i 

E *W\ = - Yl a *y* + o J2 «i«j-^(xi; xj) + e y^ l«il 

i=l 2J = 1 i=l 

we conclude that we need to minimize the following QP problem: 

Problem C.l 

min[£'*[a] — a • 6] 

subject to the constraint: 

a- 1 =0 

where the vector 6 is unknown. 

In order to understand how to deal with the fact that we do not know 6 } let us consider a different 
QP problem: 

Problem C.2 

min£ , *[al 
a L J 

subject to the constraints: 

a-1 =0 
a > 77 

a < 77* 

where the box parameters rj and rj* are unknown. 

We solve problem C.2 using the Lagrange multipliers technique for the inequality constraints, 
obtaining the following dual version of problem C.2: 



Problem C.3 



maxminfE*fal - a • (3 - 3*) + 3 ■ ri - 3* ■ n*} 

3,3* a 



subject to the constraint: 

a-1 = 

3,3* >0 

where 3 and 3* are vectors of Lagrange multipliers. 
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Notice now that the choice of the box parameters 77 and 77* uniquely determines (3 and /3*, 
and that setting 6 = (3 — (3* , problems C.l and C.3 are identical for what concerns the a 
vector: they both require to solve a QP problem in which a linear term contains unknown 
coefficients. Therefore, solving problem C.l with unknown 6 seems to be formally equivalent to 
solving problem C.3 with unknown box parameters. This suggests the following argument: 1) 
solving C.l with unknown 6 is formally equivalent to solving problem C.3 with unknown box 
parameters; 2) in absence of any information on the noise, and therefore on the box parameters, 
we could set the box parameters to 77* = —77 = C\ for some unknown C; 3) for 77* = —77 = C\ 
problem C.3 becomes the usual QP problem of SVM (problem B.l); 4) therefore, in total absence 
of information on the noise, problem C.l leads to the same QP problem of SVM, making the 
equivalence between sparse approximation and SVM complete. However this argument is not 
very rigorous, because it does not make clear how the assumptions on 77 and 77* are reflected on 
the noise vector 6. However, the formal similarity of the problems C.3 and C.l seems to point in 
the right direction, and an analysis of the relationship between 77, 77* and 6 could lead to useful 
insights on the assumptions which are made on the noise in the SVM technique. 
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